{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/peremartra/Large-Language-Model-Notebooks-Course/blob/main/2-Vector%20Databases%20with%20LLMs/2_1_Vector_Databases_LLMs.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8vPA--nKMhoQ"
   },
   "source": [
    "<div>\n",
    "    <h1>Large Language Models Projects</a></h1>\n",
    "    <h3>Apply and Implement Strategies for Large Language Models</h3>\n",
    "    <h2>2.1-Vector Databases with LLMs</h2>\n",
    "</div>\n",
    "\n",
    "by [Pere Martra](https://www.linkedin.com/in/pere-martra/)\n",
    "__________\n",
    "Models: TinyLlama/TinyLlama-1.1B-Chat-v1.0\n",
    "\n",
    "Colab environment: CPU.\n",
    "\n",
    "Keys:\n",
    "* Vector Database.\n",
    "* ChromaDB.\n",
    "* RAG\n",
    "* Embeddings.\n",
    "\n",
    "Article related: [Harness the Power of Vector Databases: Influencing Language Models with Personalized Information.](https://medium.com/towards-artificial-intelligence/harness-the-power-of-vector-databases-influencing-language-models-with-personalized-information-ab2f995f09ba)\n",
    "__________\n",
    "\n",
    "\n",
    "If you are executing this notebook on Colab you will need a High RAM capacity environment, depending on the model used.\n",
    "\n",
    "If you don't have a Colab Pro acount you can execute this notebook on kaggle, since you will get more memory from the free tier.\n",
    "\n",
    "Here yo have a version of this notebook, that uses a Dolly 3B model, that can be executed on Kaggle: [Vector Databases with LLMs-Kaggle Version](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/2-Vector%20Databases%20with%20LLMs/how-to-use-a-embedding-database-with-a-llm-from-hf.ipynb)\n",
    "__________\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rnSqcL5iOMV-"
   },
   "source": [
    "In this notebook you will see how to use an embedding database to store the information that you want to pass to a large language model so that it takes it into account in its responses.\n",
    "\n",
    "The information could be your own documents, or whatever was contained in a business knowledge database.\n",
    "\n",
    "I have prepared the notebook so that it can work with three different Kaggle datasets, so that it is easy to carry out different tests with different Datasets.\n",
    "\n",
    "![RAG Structure](https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/img/Martra_Figure_2-7.jpg?raw=true)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KDIwyhl5TTGZ"
   },
   "source": [
    "#Import Libraries.\n",
    "To start is necessaryto install some Python packages.\n",
    "\n",
    "* **sentence transformers**. This library is necessary to transform the sentences into fixed-length vectors, also know as embeddings.\n",
    "\n",
    "* **chromadb**. This is our vector Database. ChromaDB is easy to use and open source, maybe the most used Vector Database used to store embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hCD18oMGXI2p"
   },
   "outputs": [],
   "source": [
    "!pip install -q transformers==4.41.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jMtUKHm6YL5T"
   },
   "outputs": [],
   "source": [
    "!pip install -q sentence-transformers==2.2.2\n",
    "#!pip install -q xformers==0.0.23\n",
    "!pip install -q chromadb==0.4.20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rN_QB7NmYZFj"
   },
   "source": [
    "I'm sure that you know the next two packages: Numpy and Pandas, maybe the most used python libraries.\n",
    "\n",
    "Numpy is a powerful library for numerical computing.\n",
    "\n",
    "Pandas is a library for data manipulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "VuDXjIjAYgXm"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vAPpMqfWYivc"
   },
   "source": [
    "# Load the Dataset\n",
    "As you will see the notebook is ready to work with three different Datasets. Just uncomment the lines of the Dataset you want to use.\n",
    "\n",
    "I selected Datasets with News. Two of them have just a brief decription of the new, but the other contains the full text.\n",
    "\n",
    "As you are working in a memory limited environment, and you can use just a few gb of memory I limited the number of news to use with the variable MAX_NEWS.\n",
    "\n",
    "The name of the field containing the text of the new is stored in the variable *DOCUMENT* and the metadata in *TOPIC*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QYWQlIo0FzNO"
   },
   "source": [
    "# Copy Kaggle Dataset\n",
    "I used the kotartemiy/topic-labeled-news-dataset\n",
    "https://www.kaggle.com/datasets/kotartemiy/topic-labeled-news-dataset\n",
    "\n",
    "Artem Burgara. (2020). R vs. Python: Topic Labeled News Dataset, . Retrieved December 2023, from https://www.kaggle.com/discussions/general/46091.\n",
    "\n",
    "But you can ose other datasets, I encourage you to try at least one of these:\n",
    "* https://www.kaggle.com/datasets/gpreda/bbc-news\n",
    "* https://www.kaggle.com/datasets/deepanshudalal09/mit-ai-news-published-till-2023\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xJaT5EwlitXg",
    "outputId": "2831c0de-927c-417a-f343-8da83cb72015"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0U9BXNKBi9lR",
    "outputId": "8e18ddc5-9de5-41ee-8ccd-2e25642f05be"
   },
   "outputs": [],
   "source": [
    "!pip install kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GCzM-FdBjHtH"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "#This directory should contain you kaggle.json file with you key\n",
    "os.environ['KAGGLE_CONFIG_DIR'] = '/content/drive/MyDrive/kaggle'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6ynM9bwdjDyh",
    "outputId": "a5ee39d4-2276-4175-99df-bf588f04b72a"
   },
   "outputs": [],
   "source": [
    "!kaggle datasets download -d kotartemiy/topic-labeled-news-dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GCSzcqOIjlae"
   },
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "# Define the path to your zip file\n",
    "file_path = '/content/topic-labeled-news-dataset.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aUdl7-iqkttN"
   },
   "outputs": [],
   "source": [
    "with zipfile.ZipFile(file_path, 'r') as zip_ref:\n",
    "    zip_ref.extractall('/content/drive/MyDrive/kaggle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mie-ryBHL9dJ"
   },
   "source": [
    "#Loading the Dataset\n",
    "\n",
    "Although I've utilized a single dataset for the notebook, I've set it up to facilitate testing with different datasets, available on Kaggle.\n",
    "\n",
    "I selected Datasets with News. Two of them have just a brief decription of the new, but the other contains the full text.\n",
    "\n",
    "As we are working in a free and limited space, and we can use just 30 gb of memory I limited the number of news to use with the variable MAX_NEWS.\n",
    "\n",
    "The name of the field containing the text of the new is stored in the variable DOCUMENT and the metadata in TOPIC.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MeeuZ7tbPrjR"
   },
   "outputs": [],
   "source": [
    "news = pd.read_csv('/content/drive/MyDrive/kaggle/labelled_newscatcher_dataset.csv', sep=';')\n",
    "MAX_NEWS = 1000\n",
    "DOCUMENT=\"title\"\n",
    "TOPIC=\"topic\"\n",
    "\n",
    "#Just in case you want to try with a different Dataset.\n",
    "#news = pd.read_csv('/content/drive/MyDrive/kaggle/bbc_news.csv')\n",
    "#MAX_NEWS = 1000\n",
    "#DOCUMENT=\"description\"\n",
    "#TOPIC=\"title\"\n",
    "\n",
    "#news = pd.read_csv('/content/drive/MyDrive/kaggle/mit-ai-news-published-till-2023/articles.csv')\n",
    "#MAX_NEWS = 100\n",
    "#DOCUMENT=\"Article Body\"\n",
    "#TOPIC=\"Article Header\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zGy3ZO4MMY7t"
   },
   "source": [
    "ChromaDB requires that the data has a unique identifier. You can achieve it with the statement below, which will create a new column called **Id**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "id": "BPhCplVecPK6",
    "outputId": "31901f90-6717-429e-c94a-f34187f9ed76"
   },
   "outputs": [],
   "source": [
    "news[\"id\"] = news.index\n",
    "news.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "elYy8a0OTJaf"
   },
   "outputs": [],
   "source": [
    "#Because it is just a example we select a small portion of News.\n",
    "subset_news = news.head(MAX_NEWS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S8tDSbZ3MxZP"
   },
   "source": [
    "# Import and configure the Vector Database\n",
    "You are going to use ChromaDB, the most popular OpenSource embedding Database.\n",
    "\n",
    "First you need to import ChromaDB, and after that import the **Settings** class from **chromadb.config** module. This class allows to change the setting for the ChromaDB system, and customize its behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cXYYHBJzMl5n"
   },
   "outputs": [],
   "source": [
    "import chromadb\n",
    "from chromadb.config import Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tmFsTy9XOPla"
   },
   "source": [
    "Now you need to create the seetings object calling the Settings function imported previously. The object is stored in the variable **settings_chroma**.\n",
    "\n",
    "You need to inform two parameters\n",
    "\n",
    "* **chroma_db_impl**. Here you must specify the database implementation and the format how store the data. I choose **duckdb**, because his high-performace. It operate primarly in memory. And is fully compatible with SQL. The store format **parquet** is good for tabular data. With good compression rates and performance.\n",
    "\n",
    "* **persist_directory**: It just contains the directory where the data will be stored. Is possible work without a directory and the data will be stored in memory without persistece, but some cloud providers or platforms like Kaggle dosn't support that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tAbROIjCONv7"
   },
   "outputs": [],
   "source": [
    "#OLD VERSION\n",
    "#settings_chroma = Settings(chroma_db_impl=\"duckdb+parquet\",\n",
    "#                          persist_directory='./input')\n",
    "#chroma_client = chromadb.Client(settings_chroma)\n",
    "\n",
    "#NEW VERSION => 0.40\n",
    "chroma_client = chromadb.PersistentClient(path=\"/content/drive/MyDrive/chromadb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vxlh6QfiSK-p"
   },
   "source": [
    "# Filling and Querying the ChromaDB Database\n",
    "The Data in ChromaDB is stored in collections. If the collection previously exist is necessary to delete it.\n",
    "\n",
    "In the next lines, the collection is created by calling the ***create_collection*** function in the ***chroma_client*** created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G4O2UuGzW5mC"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6HOnr43oO1vG"
   },
   "outputs": [],
   "source": [
    "collection_name = \"news_collection\"+datetime.now().strftime(\"%s\")\n",
    "if len(chroma_client.list_collections()) > 0 and collection_name in [chroma_client.list_collections()[0].name]:\n",
    "        chroma_client.delete_collection(name=collection_name)\n",
    "\n",
    "collection = chroma_client.create_collection(name=collection_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "C-_0xWxQSfwv"
   },
   "source": [
    "It's time to add the data to the collection. Using the function ***add*** you should inform, at least ***documents***, ***metadatas*** and ***ids***.\n",
    "* In the **document** the full news text is stored, remember that it is contained in a different column for each Dataset.\n",
    "* In **metadatas**, we can inform a list of topics.\n",
    "* In **id** an unique identificator for each row must be informed. It MUST be unique! I'm creating the ID using the range of MAX_NEWS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xuAz6R0gSZho",
    "outputId": "a47379c8-e4a5-49fd-c055-f7be35196ca0"
   },
   "outputs": [],
   "source": [
    "collection.add(\n",
    "    documents=subset_news[DOCUMENT].tolist(),\n",
    "    metadatas=[{TOPIC: topic} for topic in subset_news[TOPIC].tolist()],\n",
    "    ids=[f\"id{x}\" for x in range(MAX_NEWS)],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IpofynmFTCKm",
    "outputId": "8fb85c27-3993-4703-970b-a130bb69290a"
   },
   "outputs": [],
   "source": [
    "results = collection.query(query_texts=[\"laptop\"], n_results=10 )\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b4PDwcbHcQqO"
   },
   "source": [
    "#Vector MAP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X8s8H-49cNmb"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xz9rF11rcP2r"
   },
   "outputs": [],
   "source": [
    "getado = collection.get(ids=\"id141\",\n",
    "                       include=[\"documents\", \"embeddings\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kLJtGzfecXAO",
    "outputId": "a7388dee-201c-42d5-99a9-1742cc506301"
   },
   "outputs": [],
   "source": [
    "word_vectors = getado[\"embeddings\"]\n",
    "word_list = getado[\"documents\"]\n",
    "word_vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hR3SaELeccOT"
   },
   "source": [
    "Once the information is on the Database you can query It, and ask for data that matches your needs. The search is done inside the content of the document. It dosn't look for the exact word, or phrase, the results will be based on the similarity between the search terms and the content of documents.\n",
    "\n",
    "The metadata is not used in the search, but they can be utilized for filtering or refining the results after the initial search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3OQoEDJ1c514"
   },
   "source": [
    "# Loading the model and creating the prompt\n",
    "TRANSFORMERS!!\n",
    "Time to use the library **transformers**, the most famous library from [hugging face](https://huggingface.co/) for working with language models.\n",
    "\n",
    "We are importing:\n",
    "* **Autotokenizer**: It is a utility class for tokenizing text inputs that are compatible with various pre-trained language models.\n",
    "* **AutoModelForCasualLLM**: it provides an interface to pre-trained language models specifically designed for language generation tasks using causal language modeling (e.g., GPT models), or the model used in this notebook ***TinyLlama-1.1B-Chat-v1.0***.\n",
    "* **pipeline**: provides a simple interface for performing various natural language processing (NLP) tasks, such as text generation (our case) or text classification.\n",
    "\n",
    "The model I have selected is [TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0), which is one of the smartest Small Language Models. Even so, it still has 1.1 billion parameters.\n",
    "\n",
    "Please, feel free to test [different Models](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending), you need to search for NLP models trained for text-generation. My recomendation is choose \"small\" models, or we will run out of memory in kaggle.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lA3c0W8i_zg4"
   },
   "outputs": [],
   "source": [
    "#!pip install -q einops==0.8.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m0OIjfQxc3rW"
   },
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline\n",
    "\n",
    "model_id = \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\"\n",
    "#model_id = \"databricks/dolly-v2-3b\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "lm_model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46Sju48Dekow"
   },
   "source": [
    "\n",
    "The next step is to initialize the pipeline using the objects created above.\n",
    "\n",
    "The model's response is limited to 256 tokens, for this project I'm not interested in a longer response, but it can easily be extended to whatever length you want.\n",
    "\n",
    "Setting ***device_map*** to ***auto*** we are instructing the model to automaticaly select the most appropiate device: CPU or GPU for processing the text generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4Gr-YtwXdCbI"
   },
   "outputs": [],
   "source": [
    "pipe = pipeline(\n",
    "    \"text-generation\",\n",
    "    model=lm_model,\n",
    "    tokenizer=tokenizer,\n",
    "    max_new_tokens=256,\n",
    "    device_map=\"auto\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "llI-0pcGjz-Y"
   },
   "source": [
    "## Creating the extended prompt\n",
    "To create the prompt you can use the result from query the Vector Database  and the sentence introduced by the user.\n",
    "\n",
    "The prompt have two parts, the **relevant context** that is the information recovered from the database and the **user's question**.\n",
    "\n",
    "You only need to join the two parts together to create the prompt sended to the model.\n",
    "\n",
    "You can limit the lenght of the context passed to the model, because you can get some Memory problems with one of the datasets that contains a realy large text in the document part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LxrmUcEGjwTc"
   },
   "outputs": [],
   "source": [
    "question = \"Can I buy a new Toshiba laptop?\"\n",
    "context = \" \".join([f\"#{str(i)}\" for i in results[\"documents\"][0]])\n",
    "#context = context[0:5120]\n",
    "prompt_template = f\"\"\"\n",
    "Relevant context: {context}\n",
    "Considering the relevant context, answer the question.\n",
    "Question: {question}\n",
    "Answer: \"\"\"\n",
    "prompt_template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f3gRd5HNkJA1"
   },
   "source": [
    "Now all that remains is to send the prompt to the model and wait for its response!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l9ZiP7QekFYS"
   },
   "outputs": [],
   "source": [
    "lm_response = pipe(prompt_template)\n",
    "print(lm_response[0][\"generated_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8n_ezyOdFnjQ"
   },
   "source": [
    "__________\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zCVJcSSzCure"
   },
   "source": [
    "# Connecting to a ChromaDB existing collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0PnkBLCWFz4z"
   },
   "outputs": [],
   "source": [
    "!pip install chromadb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Sn4jcasik4MB"
   },
   "outputs": [],
   "source": [
    "import chromadb\n",
    "chroma_client_2 = chromadb.PersistentClient(path=\"/content/drive/MyDrive/chromadb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7c5IlfB8mEHr"
   },
   "outputs": [],
   "source": [
    "collection2 = chroma_client_2.get_collection(name=collection_name)\n",
    "results2 = collection.query(query_texts=[\"laptop\"], n_results=10 )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CKhm8u77DLNu"
   },
   "outputs": [],
   "source": [
    "print(results2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yeGJrCSEJG61"
   },
   "source": [
    "# Conclusions\n",
    "A very short notebook, but with a lot of content.\n",
    "\n",
    "You have used a vector database to store information. Then move on to retrieve it and use it to create an extended prompt that you've used to call one of the newer large language models available in Hugging Face.\n",
    "\n",
    "The model has returned a response taking into account the context that you have passed to it in the prompt.\n",
    "\n",
    "This way of working with language models is very powerful.\n",
    "\n",
    "Is possible to make the model use our information without the need for Fine Tuning. This technique really has some very big advantages over fine tuning."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
